System for morphological image fusion and change detection

ABSTRACT

A system capable of efficiently fusing image information from multiple sensors operating in different formats into a single composite image and simultaneously display all of the pertinent information of the original images as a single image. The disclosed invention accomplishes this by receiving image information from multiple sensors; identifying small structure or object information in each image by setting a predefined object/background threshold; separating the object or small structure from the background or large structure in each image; blending the object information from each image into a single composite object image; blending the background information from each image into a single composite background image; combining the composite object image with the composite background image to produce a composite image containing all of the pertinent information of each of the original images, and displaying a single composite image containing all of the pertinent information contained in each of the original images.

BACKGROUND

[0001] 1. Field of the Invention

[0002] This invention relates in general to signal processing and morespecifically, to a system for fusing two or more images from differentsensors into one image.

[0003] 2. State of the Art

[0004] With the advance of image sensing technology, it has becomeincreasingly desirable to provide efficient and cost effective ways toprocess and display image information.

[0005] Today's sensing devices often provide vast quantities of diverseinformation in differing forms and configurations that human operatorsoften are unable to efficiently process visually. This situation, knownas “information overload” is often worsened when relevant information isprovided simultaneously in different formats on multiple userinterfaces, while the human operator often must focus his attentionelsewhere.

[0006] For example, a pilot must process vast quantities of informationfrom several different input devices while simultaneously piloting hisaircraft, often under conditions which are less than favorable. Forinstance, a military pilot, tasked with flying a mission at low level intotal darkness or inclement weather, must simultaneously evade hostileforces, acquire a target and accurately deliver an ordnance, whilefocusing his attention on piloting and navigating his aircraft. Thepilot cannot divert attention from the task at hand for more than a fewseconds to interpret multiple displays which provide speed, navigation,threat, targeting, weapons systems or other types of information whichmay be critical to the mission and pilot survival. Thus, relevantinformation may go unrealized with harmful and often catastrophicresults.

[0007] Specifically, one CRT in the cockpit may display images producedby an optical sensor operating in the visual spectrum, while a secondCRT in the cockpit may display images produced by a sensor sampling thesame scene operating in the IR spectrum, and a third CRT may displayimages produced by radar returns from the identical scene. Thus, toeffectively process the information from each input medium the pilotmust divert his attention from the task of flying and navigating theaircraft for a significant period.

[0008] Similarly, a physician performing laser surgery or some othersurgical procedure would need to be aware of the relationship hisinstrument has to the tissue or bone in close proximity to the areaunder repair. Cameras and CRT displays contained on the instrumentsthemselves offer some insight into the area of interest, however theycannot show bones or tissue hidden from visual inspection. X-ray, IR andother sensor/detection means are used to provide that type ofinformation, which need other display interfaces, causing the physicianto shift her attention between multiple displays.

[0009] Scientist and engineers have taken several approaches to increasethe speed and efficiency at which a human operator can receive andprocess image information from multiple sensors using multiple formats.One solution has been to use split screen displays. Split screendisplays partition a CRT into sections, each section displaying the samescene imaged by a different type of sensor. For instance one section maydisplay the scene imaged using an IR sensor, while the other sectionwill display the same scene imaged using a camera operating in thevisual spectrum, and yet another section will display a radar or x-rayimage of the same scene. While more efficient than making an operatorscan several CRTs distributed around him, this approach still requiresan operator to focus his attention on each section of the CRT andmethodically extract relevant information form each image format.

[0010] Another approach has been to employ multi-mode CRT displays.These displays normally have some type of display selection capabilitywhich allows a user to switch between different display modes, each modedisplaying, on full screen, an image of a scene produced by a differentsensor. For example, one mode may display scene images produced by acamera operating in the visible light spectrum, while another mode maydisplay the same scene imaged by an IR sensor while yet another mode maydisplay the same scene imaged by a radar or x-ray unit. This approachreduces the number of CRT displays necessary for displaying the imageinformation, however it requires an operator to select the display modeand to focus attention on multiple displays modes to extract relevantinformation unique to each display mode (sensor).

[0011] Methods and systems are known for fusing image information frommultiple sensors operating in different formats into a single compositeimage simultaneously displaying relevant information from each sensor.

[0012] However, known methods of fusing multiple images into a singlecomposite image generally employ linear filter approaches followed bysimply adding the images together pixel by pixel. Conventional linearfiltering approaches create a new image by calculating a weightedaverage of the pixel intensities in the local area of the originalimage. In linear filtering his is referred to as a convolutionoperation. A small mask representing the “weights” to be used in theaverage is moved across an intensity plot of the scene. Each pixelcovered by the mask is multiplied by the appropriate weighting factor.The sum of all of the weighted pixels values becomes the new pixel valuein the new image. FIG. 1, is an example of such an intensity plot 100,in which the image intensity is plotted across a selected horizontalline of an scene. The plot is across a selected horizontal line of ascene, thus the y coordinates are not apparent in FIG. 1. The peaks andvalleys in the intensity plot are representative of the changes inintensity as the sensor samples a scene. A rapid change in intensity,such as shown by event 102, or event 108 suggests some change in thescene such as moving from background to an object or from object tobackground.

[0013] Generally, high frequency structure (that is, a pronounced changein the intensity over a short distance, or a small area of pixels), isassociated with objects within a given image while low frequencystructure (that is, a change covering a larger distance or area) isassociated with the background.

[0014] Prior image fusion methods use conventional linear filtering toseparate high frequencies, from the background by tracing the movementof a convolution mask (for example, element 104, represented as awindow) as it slides over the intensity plot of the scene as shown inFIG. 1. Conventional linear filters remove high frequencies from theimage scene to produce a new background or low frequency image by usinga weighted average of the input image intensity 100 calculated overelement 104 to produce the intensity plot as shown in FIG. 2. In thiscase element 104 is a convolution filter or convolution mask containingthe weighs used in the local average.

[0015] The difficulty in using linear filtering techniques is thatactual objects in a scene or image are composed of many differentfrequencies. Both large and small object contain high frequencies (e.g.,the edges). Just removing high frequencies from an image will removesmall objects or structure, but will also blur the edges of the largeobjects. Thus, modifying the frequency content of an image is a poormethod for modifying the content of an image.

[0016] The use of a linear filter, allows the intensity of a smallobjects to affect the intensity values of local areas, causing residual“bumps” 202 and 208 to be left at the location of a small objectfiltered from a scene as shown. This practice of employing an averagecan cause an undesirable blur 210 at places on the image where there isa pronounced change in intensity, such as a change in the background ofthe scene 110 or the onset of an object contained therein as shown by102 and 108 of FIG. 1. Thus conventional linear filtering tends to blurthe separation between objects and the background and is thusinefficient when tasked with extracting objects from imagery. As aresult, the use of conventional linear filters in image fusionapplications has been limited in using the local high frequency contentof the imagery as a parameter in determining the intensity of the fusedimage at that location. Object identification is not attempted. Thisblurring effect common to linear filtering techniques also makes thesystem vulnerable to high frequency noise or fine grained structurepatterns within the imagery. This effect also produces color stabilityproblems when a scenes processed using linear filter techniques aredisplayed in color. The effect is magnified when the scene is changingor when in employed in a dynamic environment.

SUMMARY OF THE INVENTION

[0017] The present invention is directed to a structure or objectoriented method and system for efficiently fusing image information frommultiple sensors operating in different formats into a single compositeimage simultaneously displaying all of the pertinent information of theoriginal images. The present invention finds objects and structurewithin the various images which meet a very general user defined sizeand shape criteria. It then inserts these objects into the fused image.The background of the fused images is obtained by the combination of thebackgrounds of the input images after the objects have been removed. Thevarious objects can be intensity or color coded based on theirintensities in the source image from which each object came.

[0018] The present invention employs morphological filters or shapefilters to process multiple signals produced by one or more imagingdevices, separating the background image signals from the object imagesignals produced by each device on the basis of object size or structureorientation (independent of its intensity), thus eliminating theblurring normally associated with conventional linear image processing.

[0019] Morphological filters do not use a convolution mask or a weightedaverage of the local pixels. Instead morphological filters use a“structuring element’ which defines the size (and shape) of theintensity profiles that the user wants to use as a definition of thesize and shape of an object.

[0020] Once the objects are removed from the backgrounds the variousimage signals are combined into one or more composite images, allowing auser to display selected information from several scenes as a singlescene or image.

[0021] According to one aspect of the invention a system employingnonlinear morphological or shape filters provides for the fusionselected images from different sensors sampling the same orsubstantially the same scene into one or more composite images.

[0022] According to another aspect of the invention, images from thesame sensor sampling a scene at different times is fused into a singleimage. This image may be presented on a single display.

[0023] According to yet another aspect of this invention, via the use ofnonlinear filters, objects contained in an image scene are distinguishedfrom the background structure of the scene based on size or structure.

[0024] According to another aspect of the present invention, images fromdifferent sensors are fused into a composite image. The images fromdifferent sensors may be color coded and selectively presented as asingle image scene on a means for display.

[0025] According to yet another aspect of the present invention selectedimages from a single sensor sampling a scene at different times arefused into a single image. The images may be color coded based on thepoint in time sampled and presented as a single image scene on a meansfor display. dr

BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Other objects and advantages of the present invention will becomeapparent to those skilled in the art upon reading the following detaileddescription of preferred embodiments, in conjunction with theaccompanying drawings, wherein like reference numerals have been used todesignate like elements, and wherein:

[0027]FIG. 1 shows an example intensity plot across a selectedhorizontal line of an image scene with a size element beneath the plot.

[0028]FIG. 2 shows an attempt to remove the small structure (objects)from the scene using (prior art) conventional linear filter.

[0029]FIG. 3 shows the intensity plot of FIG. 1 with the small structureremoved from the scene using morphological filters.

[0030]FIG. 4 shows a block diagram of a system for morphological imagefusion and change detection.

[0031]FIG. 5 shows a flowchart for a morphological image fusionalgorithm.

[0032]FIG. 6 illustrates an example series of intensity plots 6(a)-6(g)received and produced by the example circuit of FIG. 4.

[0033]FIG. 7 illustrates vertical erosion process performed on a 3 pixel×3 pixel area of an image scene.

[0034]FIG. 8 illustrates horizontal erosion process performed on a 3pixel ×3 pixel area of an image scene.

[0035]FIG. 9 shows a series of intensity plots in which small positivecontrasting objects are removed by erosion operations followed bydilation operations.

[0036]FIG. 10 shows a series of intensity plots in which small negativecontrasting are removed by erosion operations followed by dilationoperations

[0037]FIG. 11 shows a scene with a single isolated pixel object and amulti pixel cross image.

[0038]FIG. 12 illustrates and example of a multi dimensional nonlinearcross filter.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0039] In the following description, specific details are set forth inorder to provide a through understanding of the invention. However, itwill be apparent to those skilled in the art that in invention can bepracticed in other embodiments that depart from these specific details.

[0040] The present invention is directed to a system which employsmorphological filters in image processing. Morphological filters can beimplemented in various ways such as maximum/minimum operations,dilate/erode operations, or order statistic operations such as medianfilters.

[0041]FIG. 4 shows a block diagram of a system for morphological imagefusion and change detection 400, employing morphological filters toprocess image signals from one or more scenes by separating objects orstructures contained in each image signal from the background objects orstructures of each image signal, based on size and orientation. Thebackground and object images are then selectively fused into a compositeimage displaying desired scene information extracted from each imagesignal.

[0042] Small structure, or a significant change in the intensity over agiven distance in the image is generally associated with objects, whilelarge structure is associated with the background. Therefore, byremoving small structure from the image signal the filter is able todistinguish and separate the objects from the background.

[0043] Referring again to FIG. 1, morphological filters create a new“large structure” image by tracing the movement of structuring element104 as it slides below intensity plot 100, and filtering out any event,large structure, that covers a smaller area than structuring element104. Generally, large structure contains not only the low frequencies inthe image but contains also the high frequencies associated with theedges of the large structure.

[0044] Structuring element 104 defines a preselected maximum “target”structure size. This maximum “target” structure size, is the maximumsize of an event in the horizontal direction, in pixels, which thefilter will consider an object. Any transition in intensity requiringmore pixels (a greater area) than defined by the dimensions of thestructuring element will be considered by the morphological filter to bea part of the background of the scene. If the structuringelement=2(n)+1, then an event 2(n) pixels or smaller will be classifiedas an object. The morphological filter's resolution, it's ability todistinguish objects from the background, is thus determined by thedimensions, in pixels, of the structuring element used by eachmorphological filter. Thus by distinguishing objects from background onthe basis of size, objects can be separated from an scene without theblurring effect common in prior methods.

[0045]FIG. 3 shows the intensity plot of FIG. 1 with the small structureremoved from the scene using morphological filters in place of thelinear filters. If structuring element 104 does not “fit” up into aregion of the intensity profile then it is considered to be an “object”.If it does fit, it is considered to be part of the background. Asdepicted in FIG. 3 residual “bumps” 202 and 208 and the blurring effect210 have been eliminated. (This technique would also have to be repeatedfor negative contrast objects having an intensity lower than the localbackground. Structuring element 104 would have to not fit “down” into anobject.)

[0046] Referring now to FIG. 4 which shows an example embodiment of asystem for morphological image fusion and change detection, system 400,is coupled to at least one sensing means (not shown). The sensing meansmay be a single sensor sampling a scene at different times, or an arrayof sensing devices which image a scene in IR, visible light, radar,sonar, magnetic or electric field, chemical, conventional orunconventional means or a combination thereof, simultaneously samplingthe same or substantially the same scene. Image signals I1 and I2, 402and 404, are the signals produced by sensing devices. One skilled in theart will appreciate that the sensing means is not limiting in thisinvention and that I1 and 12 are image signals produced by sensing meanssampling a similar or substantially the same scene.

[0047] In the preferred embodiment the received image signals 402, 404are coupled into morphological type filters 406 and 408, which separatethe objects from the background via use of a structuring element.

[0048] Each morphological filter 406, 408 is coupled to a differencecircuit 410, 412, which receives the filtered signal B1(x) and B2(x)from the morphological filters. Each morphological filter 406, 408 isalso coupled to blend circuit 416, which receives the filtered imagesignal B1(x) and B2(x) produced by the morphological filter.

[0049] Difference circuit 410 is coupled to blend circuit 414. Blendcircuit 414 receives the output of difference circuit 410, image signalΔS1 420 as input.

[0050] Difference circuit 412 is also coupled to blend circuit 414.Blend circuit 414 also receives the signal produced by differencecircuit 414, image signal ΔS2 422. Blend circuit 414 is also coupled tosumming circuit 418 which receives the output from blend circuit 414 andthe output from blend circuit 416. The output of summing circuit 418 iscoupled out of the system 400.

[0051] With reference also to FIG. 5, which illustrates the flowchart500 of the operation of the example embodiment of a system morphologicalimage fusion 400 as illustrated in FIG. 4, image signal (I1) 402 isreceived into the system from sensing means 502. The morphologicalfilter, in this example large structure pass filter 406, then removesthe high frequency structure from image signal I1 as shown in block 506.

[0052] Large structure, or a change in the intensity plot of a selectedportion of an image signal is generally associated with objects whilesmall structure is associated with the background. As discussed above byremoving the small structure from the image signal, the objects areseparated from the background.

[0053] The size threshold of the large structure pass filter used in thecurrent example embodiment is determined by the size and shape of thestructuring element. As the structuring element becomes smaller, anevent (a change in the image signals intensity) must occupy a smallerarea in order to be considered an object and filtered from the scene.This results in a change in the filters size threshold. Thus an event,an intensity change in the image signal, must occupy a smaller area onthe intensity plot to fall outside the structuring element sizethreshold of the filter and be removed from the scene.

[0054] Referring again to the figures, difference circuit 410, receivesfiltered signal B1(x) 424, with high frequency structure, removed fromthe image signal. This filtered or truncated image signal is associatedwith the background images of the scene sampled by sensor 1. Differencecircuit 410 compares this background image signal with the unfilteredimage signal I1 402, as shown in block 510, the difference of whichrepresents the object images ΔS1 420 contained in the scene sampled bysensor 1.

[0055] In the example embodiment, this filtering process is repeated ona second image signal. Image signal 12 is received by the filter circuitfrom the sensor and large structure pass filter 408 or another type ofnonlinear filter employs a structuring element to filter small structurefrom the scene 508. The filtered signal B2(x) is coupled into blendcircuit 416, and is combined with the filtered signal B1(x) to form acomposite background image ΣB(n), as shown in 516 of FIG. 5. Thiscomposite background image may be a combination of each background imageproduced by each filter, a combination of selected background images orthe composite may contain a background image produced by a singlefilter.

[0056] The filtered signal B2(x) is also coupled into difference circuit412, which receives background image B2(x) and compares this backgroundimage signal with the unfiltered image signal (12) 404, as shown inblock 512, the difference of which represent the object images ΔS2 422contained in the scene sampled by sensor 2.

[0057] Blend circuit 414 receives object image signals ΔS1 and ΔS2,produced by difference circuits 410 and 412 respectfully, and mayselectively fuse or combine these signals into a composite object image,ΣΔSn, 514 containing object images from each of the scenes sampled.

[0058] These signals may then be selectively combined with compositebackground scene ΣB(N) 518 and displayed as a single image scene 520.

[0059]FIG. 6 illustrates an example series of one dimensional intensityplots 6(a)-6(g) received and produced by the example circuit of FIG. 4.Assume for the purposes of this example that the I1 and 12 image signalsreceived from the sensor means are represented by S1 and S2 in the FIG.6. FIG. 6(a) represents (S1) image 402 of a scene sampled by a sensoroperating in the visible light spectrum as received from sensor 1, 402in the example circuit of FIG. 4. The S1 image, 602, includes background(B1) and object (ΔS1) images. FIG. 6(b) represents the S2 image 608 ofthe same scene sampled by a sensor operating in the infrared lightspectrum as received from sensor 2, 404 in the example circuit of FIG.4. The S2 image, 608, includes background (B2) and object (ΔS2) images.When morphological filter 406 receives (S1) image 402, the morphologicalfilter removes small structure, events smaller than the structuringelement, to produce background (B1) image 606 sampled by sensor 1 asshown in FIG. 6(d). The difference between image (B1) 606 and (S1) image602 represent ΔS1 object images 604 detected in the visible lightspectrum by sensor 1 as illustrated in FIG. 6(c). Likewise, filter 408receives image (S2) 404, morphological filter removes small structure,events smaller than the structuring element, to produce (B2) backgroundimage 612 sampled by sensor 2 as shown in FIG. 6(f). The differencebetween image (B2) 612 and image (S2) 608 represent object (ΔS2) images610 detected in the infrared light spectrum by sensor 2 as illustratedin FIG. 6(e).

[0060] Once image scenes sampled by each sensor are separated intobackground and image signals, the system builds a composite scene fromselected image and background signals. This composite image scene may beselected objects and backgrounds from each sensor, objects alone or anycombination of object images and background images separated based onsize and selectively fused into a single image signal which may bedisplayed on a single output device. The selection may be automatic, orbased on image size, position, object orientation, or other userselected variable. FIG. 6(g) displays a composite image containing bothobject images and background images from both S1 and S2 sensors.

[0061] In a preferred embodiment, morphological filters are implementedusing basic erosion/dilation operations, also known as min/maxoperations. During erode/dilate operations, the system slides thestructuring element under the intensity plot of the image, processingthe section of the image over the structuring element pixel by pixel.The erode/dilate operation consist of the morphological filter lookingsequentially at each pixel in the image and comparing that pixelsintensity with the intensities of the neighboring pixels. Thisstructuring element is moved one pixel at a time, with erode or dilateoperations being performed on each pixel covered by the mask until everypixel in the image scene has been processed. Once each pixel in theimage scene is sampled and eroded or dilated the system will makesubsequent passes on the convolved image produced by the previouserosion or dilation operation. Depending on whether the system isperforming an erosion or a dilation, the intensity of the pixel beingsampled will be replaced by the minimum value of the neighboring pixelwithin the area defined by the structuring element, or the maximum valueof the neighboring pixels, respectively, within the area defined by thestructuring element.

[0062] In another embodiment, the image processing is expanded to twodimensions. In two dimensions the structuring element becomes a twodimensional structuring element thus the shape, in addition to the sizeof the structuring element determines the size and the shape of theobjects that are removed from the image. Hence, in two dimensions thetarget structure becomes any two dimensional intensity profile intowhich at least one long thin structuring element will “fit”.

[0063] The system processes the image by making multiple passes, erodingand dilating the pixel in the center of the structuring element based onneighboring pixels with horizontal, vertical and/or diagonalorientations. As discussed above, the system will use the structuringelement to make several scans of the image, eroding on one or morepasses, and dilating on one or more subsequent passes. The system alsomay alter the order of dilating and erosion operations to filterdifferent types of objects from an image. For example if the erosionoperation is followed by a dilation operation small, negativecontrasting objects will be removed from the image. If the dilationoperation is performed first, followed by erosion, small positivecontrasting objects will be removed.

[0064]FIG. 7 illustrates a 3×3 pixel area of an image scene sampled byone of the above mentioned sensors. As discussed earlier, the size ofthe area sampled at one time by the morphological filter is a functionof the size of the structuring element. In the FIG. 7 example, the areasampled is a 3×3 pixel area, hence assume for purposes of this examplethe structuring element selected is a 3×1 pixel vertical rectanglestrip. Therefore, any event requiring 3 pixels or more in the verticaldirection will be classified as part of the background.

[0065] When performing erosion/dilation operations, the system selectspixels sequentially, and compares the intensity of the selected pixel tothe intensities of the surrounding pixels. Referring to FIG. 7, eachblock represents a pixel and the value illustrated in each blockrepresents the intensity of that particular pixel. If, for example, onewanted to filter object images having positive contrasting intensities,the processor would first perform erosion operations, followed bydilation operations. With continued reference specifically to FIG. 7 inperforming vertical erosion the processor selects the pixel in thecenter of the fame, in this case having an intensity of 10 magnitude,and compares it to its neighboring pixels in the vertical plane, shownhere as having intensities of 5 and 12 magnitude. The processor thenperforms an erosion operation, by adjusting the value of the sampledpixel's intensity to correspond with intensity of its lowest verticalneighbor within the sampled structuring element mask. Thus, the erodedpixel value would be adjusted from 10 to 5 magnitude. Upon completion ofthe erosion operation on that pixel, the processor moves the structuringelement mask by one pixel, and repeats the operation for the next pixeluntil all of the pixels in the image's vertical plane have beenselected. The processor then takes the eroded image and performs adilation operation. Dilation is similar to erosion except that thevalues of the selected pixel is adjusted to correspond with themagnitude of the largest neighboring pixel.

[0066] In yet another variation the erode/dilate operations may berestricted to pixels neighboring the center pixel having a specificorientation in relation to the center pixel. For example, theneighboring pixels which are compared to the center pixel can berestricted to only certain neighboring locations, for example those onlyin horizontal direction. FIG. 8 illustrates an example of horizontalerosion. The original image 700, shows a magnitude of 10 for the centerpixel which is being convolved. In horizontal erosion the magnitude ofthe pixels directly adjacent to the center pixel, in the horizontaldirection are used to erode the center pixel. The eroded image mask 820shows the magnitude of the center pixel minimized based on the smallerof the two horizontal adjacent pixels. By repeating this operation “N”times, the resultant pixel value at any location is equal to the minimumpixel value in the original image within a range of plus or minus Npixels in the horizontal direction (or within the area defined by thestructuring element). The N 3×1 pixel erosions in the horizontaldirection followed by N3×1 dilations in the horizontal direction producethe effect of a structuring element of length 2N+1 pixels in thehorizontal direction. This structuring element will remove positivecontrast structure. To create the same size structuring element toremove negative contrast structure the order of the dilates and erodesare reversed. (I.e., first perform N dilations and follow with Nerosions.)

[0067] Horizontal dilation operations are similar to erosion except thatthe center pixel is replaced by the maximum value of the specificneighborhood pixels rather that the minimum values.

[0068] In addition to horizontal erosion and dilation, vertical anddiagonal erosion and dilation operations are possible. In verticalerosion/dilation the center pixels intensity is compared to the directlyadjacent pixels in the vertical direction. In diagonal erosion/dilationthe center pixel's intensity is convolved based also on the directlyadjacent pixels, however this time in each of the diagonal directions.

[0069] As discussed above, the order and the orientation of the dilationand erosion operations may be combined to remove structure having aspecific characteristics. For example, if erosion operations arefollowed by dilation operations, small objects having positive contrastare removed from a scene. FIG. 9 displays a series of intensity plots inwhich small positive objects are removed by erosion operations followedby dilation operations. Referring to FIG. 9, the original image 900,contains several events having, positive 902 and negative 904 contrast.The first pass, using a structuring element three pixels across,horizontal erosion operations are performed, the resulting intensityplot 910 shows the change in the plot, resulting from the erosionoperation. Specifically, the removal of some structure designated asobjects, based on size, is shown as a broken line. A dilation operationis then performed on the eroded intensity plot 910, producing theintensity plot 920. Note that positive contrast events 902 smaller thanthe structuring element, thus designated as objects have been filtered,while the negative contrasting event 904 remains.

[0070] In a second example erosion operations can be preceded bydilation operations to remove small objects having a negative contrast.FIG. 10 displays a series of intensity plots in which small negativeobjects are removed by erosion operations followed by dilationoperations. Referring to FIG. 10, the original image 1000, containsseveral events having, positive 1004 and negative 1002 contrast. In thefirst pass, using a structuring element three pixels across, horizontaldilation operations are performed, the resulting intensity plot 1010shows the change in the plot, resulting from the dilation operation.Specifically, the removal of some structure designated as objects, basedon size are shown as a broken line. An erosion operation is thenperformed on the eroded dilated plot 1010, producing intensity plot1020. Note that the negative contrast objects 1002 smaller that thestructuring element have been removed, while the positive events 1004remain.

[0071] The size and shape of the structuring element need not be limitedto 3×1 horizontal or vertical elements as described previously. Thestructuring could be any 2 dimensional size and shape as long as thedilate/erode or minimum/maximum is performed over all of the pixelswithin the shape. For example, the structuring element could be a threeby three square. A dilate would consist of creating an image with thepixel value at any location equal to the largest pixel within the areadefined by the particular shape of the structuring element. If the 3×3structuring element is used, then a background object or structure isany 2 dimensional intensity distribution which “fits’ within the a 3×3square.

[0072] The use of a sized structuring element in determining whatstructure to filter from a scene allows one to filter not only discreteobjects but also linear structure greater than a specified length. Thisstructure may or may not be part of an actual object.

[0073] In yet another example, consider the image shown in FIG. 11, inwhich a single isolated pixel object 1108 is desired to be extracted andfused into a new image. In this case the background is any structuregreater than one pixel across in any direction. Consider, further thecase where the image 1100 contains a large cross 1102 which is NOTdesired to be in the fused image.

[0074] Each arm of cross 1102 is one pixel thick but many pixels long.If the previously described procedure were followed such as a horizontalerode/dilate, the single pixel 1108 would be removed but also thevertical arm of cross 1102 would be removed. This would result in thevertical arm of cross 1102 appearing in the final fused image- which isnot the desired result. If a vertical erode/dilate were used then thehorizontal arm of cross 1102 would appear in the fused image—alsoundesirable.

[0075] Prior art linear approaches are ill-equipped to deal with thistype of scene, providing poor performance and marginal resolution.

[0076] Use of the disclosed system for image fusion employingmorphological filters in a size based approach solves this problemallowing one to define the background as any intensity structure whichis removed by a one pixel thick linear structuring element 1104 of thedesired length but at any orientation. This approach prevents long thinstructure from “leaking through” into the fused image. The disclosedapproach employs at least a horizontal and a vertical structuringelement at a minimum. In a preferred embodiment the system performs atleast two other diagonal orientations for improved performance. (I.e.,diagonal of lower left to upper right and upper left to lower right.)Each structuring element 1104 is applied individually to the image usingthe dilate/erode sequence discussed above. Also, each structuringelement 1104 is applied individually to the image using the erode/dilatesequence. The results of each of the structuring element applicationsare then combined.

[0077] In two dimensions positive background structure is defined as alinear structure in which the structuring element of the (proper length)but any orientation “fits’ within the 2 dimensional intensity profile ofthe image. Alternately, negative background structure is defined as alinear structure in which the structuring element of the (proper length)but any orientation “fits’ within the 2 dimensional intensity profile ofthe image. By selecting various combinations and subtraction of thesespatial filterings with different orientations and contrasts, one canobtain a fused image containing structure of the desired size.

[0078] This method “fuses” linear structures rather than objects. Mostobjects are composed of various combinations of linear structures.

[0079] In general the large structure positive background image may beobtained by dividing all of the structuring element processed imagesinto two groups. One group containing the processing which started witha dilate followed by erodes, and the other group which started witherodes followed by dilates. Each group contains a processed input imagefiltered with a structuring element at each angular orientation, e.g.,horizontal, vertical, and diagonals.

[0080] Next, two resultant images are generated. The positive contrastbackground structure is obtained by taking the maximum pixel value ateach location from each of the angular orientations for the filteringwhich started with an erode operation. The negative contrast backgroundstructure is obtained by taking the minimum pixel value from all of theangular orientations which started with a dilate operation.

[0081] Finally, the small structure from all of the images to be fusedis obtained by subtracting the positive and negative contrastbackgrounds from the original image. This combination of the variousorientations of linear 1×N structuring elements is called a “CrossFilter” and is part of the method for processing the image signals. Thismethodology fused imagery is based on linear structural content withinthe imagery. The structure need not be part of an actual object. Thisapproach based on structure eliminates many of the spurious artifacts ofother image fusion approaches that base the fusion on very small localdetails within the imagery. Small changes in the fine details result inlarge variations in the quality of the fused image.

[0082] The erosion/dilation methods described above can be combined intoone or a series of nonlinear filters which, by virtue of the sequence ofand/or orientation of the erode/dilate operations can remove objects orstructure having particular intensity characteristics in two dimensions.The example morphological filter, as shown in FIG. 12, may beconstructed to remove positive and negative contrast objects from abackground scene.

[0083] Referring to FIG. 12, the cross filter is constructed to use 2dimensional processing to remove both positive and negative contrastingobjects from image signal, I. Cross filter 1200 comprises two processingbanks, 1210, 1220 each bank employing a combination of erosion anddilation methods to remove negative or positive contrasting objects formimage signal I. First processing bank 1210, employs, erosion operationsfollowed by dilation operations in two dimensions. The image signal maybe simultaneously processed with horizontal 1212, vertical 1214, and twodiagonal 1216, 1218 dilations, each of which are followed by thecorresponding erosion operations. The processed signals from eachcombination of dilation preceding erosion operations are then combinedinto a single signal (LFh), 1230 representing only large positivecontrasting structure.

[0084] A second processing bank 1220 also receives the image signal,simultaneously processing the image using dilation operations followedby erosion operations. Similar to first processing bank 1210, the imagesignal may be simultaneously processed with horizontal 1222, vertical1224, and two diagonal 1226, 1228 erosions, followed by correspondingdilation operations. The processed signals from each combination oferosion preceding dilation operations are then combined into a singlesignal (LFc),1240 representing the large negative contrasting structure.

[0085] The morphological filter then combines the LFc and LFh signalsproduced by processing banks 1210, and 1220, into a single signalcontaining large negative and positive contrasting structure. Thiscombined signal has the small structure, objects, removed and reflectsthe background of the image scene I. This twin bank cross filter 1200may be substituted as filter 406 in the block diagram of FIG. 4. System400 then compares the large structure background image with the imagescene I, the difference ΔS1 reflecting the objects within the imagescene I.

[0086] In yet another embodiment, the system may employ a singlemorphological or other non-linear sensor which will capture multipleimages of the same scene at different times. Each of these signals canbe processed and compared with the original image scene to detect anychange in the object or background images. Similarly, the signals neednot be simultaneously processed using multiple processors. A system maysequentially process images using a single processor.

[0087] In yet another embodiment, multiple arrays of sensors may bebundled into a single sensor package, producing a multitude of signals,which may be simultaneously processed by a multitude of morphologicalfilters or sequentially processed by a single morphological filter.

[0088] In yet another embodiment the object images and background imagesfrom different sensors can also be color coded for display, for exampleinfrared background images may be displayed in red, infrared objects maybe displayed in orange, visible light background images may be displayedin green, and visible light objects in blue, etc.

[0089] For example, all objects which appeared in source image A but notin source image B could be colored green. All objects which appeared inboth source images could be colored blue, etc. (Note that portions of anobject made could be displayed in different colors if the originalobject was not completely captured in all of the sensor images.) Inaddition, the color intensity or saturation can be proportional to theobjects intensity. Also, objects could be eliminated or coloreddifferently if their intensity is below a certain user definedthreshold.

[0090] The superior resolution and accuracy of structure removed via theuse of the disclosed system employing morphological filters alsoprovides a solution to the color stability problem common in linearmethods. Since the object and background is defined based on structure,rather than a weighted average of the local pixels, the blurring effectand the related color stability problem is solved.

[0091] A typical color coding scheme would be to color only the objectsas described above while displaying the background as monochrome. Thusthe objects stand out but the remaining background is still visible.

[0092] The foregoing descriptions of the preferred embodiments areintended to be illustrative and not limiting. It will be appreciated thenumerous modifications and variations can be made without departing fromthe spirit or scope of the present invention.

1. A method for processing two or more image signals, each of said image signals produced from sampling substantially the same scene, comprising the steps of: a) receiving a first and second image signals from at least one sensing means; b) designating background images and object images in each of said two or more image signals on the basis of their size; c) separating said background images from said object images in said first and second image signals;
 2. A system for fusing two or more image signals comprising: nonlinear filtering means for filtering background images from object images in each of said image signals; means for blending at least one of said object images filtered from each of said image signals into a single composite object image; and means for blending said single composite object image with a background to produce an output signal.
 3. A system for fusing two or more image signals comprising: at least a first and a second signal input means; at least a first and a second non linear filter, said first nonlinear filter being coupled to said corresponding first signal input means and said second nonlinear input filter being coupled to said corresponding second signal input means, said first and second nonlinear filters producing filtered image signals; a signal coupler, said signal coupler coupled to each of said at least two nonlinear filters, blending said filtered image signals produced by each of said at least two nonlinear filters to produce a composite image signal; an output display, said output display receiving said composite image signal from said signal coupler.
 4. A system for fusing two or more image signals comprising: at least a first and a second signal input means; at least a first and a second non linear filter, said first nonlinear filter being coupled to said corresponding first signal input means and said second nonlinear input filter being coupled to said corresponding second signal input means, said first nonlinear filter receiving a first input image signal through said first input means producing a first background image signal and a first object image signal, said second nonlinear filter receiving a second input image signal through said second input means producing a second background image signal and a second object image signal; a first signal coupler, said signal coupler coupled to each of said at least two nonlinear filters, blending said first and second object image signals and at least one of said background image signals into a composite image signal; an output display, said output display receiving said composite image signal from said signal coupler displaying said composite image signal.
 5. The system of claim 4 further comprising a second signal coupler, said signal coupler coupled to each of said nonlinear filters and said first signal coupler, said second coupler receiving said first and said second background image signals from at least two of said nonlinear filters, blending said first and second background image signals into a composite background image.
 6. The method of claim 1 wherein said designating step is performed using a structuring element.
 7. The method of claim 6 wherein said filtering step is performed using at least one morphological type filter.
 8. The method of claim 6 wherein said designating step is preformed by selecting structure within the image signal that is larger than said structuring element and designating said selected structure as background images.
 9. The method of claim 6 wherein said designating step is preformed by selecting structure within the image signal that is smaller than said structuring element and designating said selected structure as object images.
 10. The method of claim 6 wherein said filtering step is preformed by employing erode/dilate operations.
 11. The system of claim 4 wherein at least one of said nonlinear filters is a morphological filter.
 12. The system of claim 11 wherein at least one of said morphological filters is a cross filter.
 13. The system of of claim 12 wherein at least one of said cross filters erosion operations precede dilation operations.
 14. The system of of claim 12 wherein at least one of said cross filters dilation operations precede erosion operations.
 15. The method of claim 1 wherein said object images in said image signals are selectively combined into a composite object image.
 16. The method of claim 1 wherein said background images from said image signals are selectively combined into a composite background image. 